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When Schools Stay Open Late: 

New Findings From the National Evaluation 

Executive Summary 



After-school programs have grown rapidly in recent years, spurred by rising employment 
rates of mothers, pressure to increase academic achievement, and concerns about risks to 
children who are unsupervised during after-school hours. The percentage of public schools 
offering “extended day” programs (which include before- and after-school programs) more than 
tripled from 1987 to 1999, from about 13 percent to 47 percent. 

The federal government’s investment in after-school programs has grown rapidly as well. 
Funding for the 21st Century Community Learning Centers program, created in 1994, rose from 
$40 million in 1998 to $1 billion in 2002. The program now provides funding to 2,250 school 
districts to support school-based programs in 7,000 public schools. 

Some studies of after-school programs have found that these programs increase academic 
achievement and student safety, as well as reduce negative behaviors such as drug and alcohol 
use. However, other studies have found that after-school programs have no effect on — and even 
worsen — certain outcomes, leading to debate over whether the evidence supports increased 
investment in after-school programs. 

In 1999, the U.S. Department of Education contracted with Mathematica Policy Research, 
Inc., and Decision Information Resources, Inc., to evaluate the 21st Century Community 
Learning Centers program. The evaluation team collected student outcome data in five areas: 
after-school supervision, location, and activities; academic performance and achievement; 
behavior; personal and social development; and safety. Because the purpose of the 21st Century 
Community Learning Centers program by law is safe and drug-free learning environments for 
students that support academic achievement, this evaluation focused on student and school 
outcomes. It did not explore the full range of parental needs and satisfaction that might be 
affected by the availability of after-school programs. It did collect parent outcome data on 
involvement in school activities and employment status. 

In its first year of data collection, the team gathered data for roughly 1,000 elementary 
school students in 18 schools in 7 school districts, and 4,300 middle school students in 
61 schools in 32 school districts. The elementary study was based on random assignment, in 
which outcomes of students assigned to the program were compared with outcomes of students 
not assigned to the program. The middle school evaluation was based on a matched-comparison 
design, in which outcomes of students who participated in programs were compared with 
outcomes of similar students who did not. bindings from these data were presented in the 
study’s first report (hereafter referred to as the “first report”), which was released in Lebruary 
2003. 

Lor the second year of data collection, researchers gathered additional data in two ways. 
Lirst, they added more elementary school programs and students. Second, they followed middle 
school students for a second year, which enabled the evaluation to explore whether there were 
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outcome differences after two years. The results are summarized in this new report, which 
contains findings from this second year of data collection. A third report will analyze impacts 
for elementary students after two years. 



Key Findings From the Second Year 

The findings from the second year of the study are generally consistent with those from the 
first year. Specifically, the study found 



• Supervision by Other Adults Increased. Students in programs were more likely to be 
with adults who were not their parents after school and less likely to be with their 
parents or older siblings. 

• Self-Care Was Unaffected. Participation in programs had no effect on whether 
students were in self-care (so-called latch-key children) after school. Multiple 
definitions of self-care were analyzed with similar results. 

• Few Impacts on Academic Achievement. Programs did not affect reading test scores 
or grades for elementary students. Grades for middle school students in programs 
were higher in social studies relative to the comparison group but not in English, 
mathematics, and science. Programs did not increase whether elementary or middle 
school students completed their homework. Middle school students in programs 
missed fewer days of school and were more likely to aspire to attend college. 

• Elementary Students Felt Safer. Elementary students in after-school programs 
reported feeling safer during after-school hours. Middle school students did not 
report feeling safer. 

• Mixed Evidence on Negative Behavior for Middle School Students. Some estimates 
pointed to higher levels of negative behaviors for middle school students, while 
others indicated no differences between treatment and comparison groups. 

• Some Impacts on Parent Outcomes. Parents of participating elementary school 
students were more likely to report that they attended school events. Other measures 
of parent involvement did not increase. There was some evidence that programs 
increased whether mothers of elementary students worked or looked for work. 
Involvement of middle school parents did not differ between the treatment and 
comparison groups. No employment difference was observed for mothers of middle 
school students. 

• Few Impacts on Developmental Outcomes. Elementary students were more likely to 
report helping other students after school. They were no more likely to report being 
able to work with others on a team, believe the best of other people, or set goals and 
work to achieve them. Middle school students showed no differences in these 
outcomes. 
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• Low Middle School Attendance in Second Year. Two attendance patterns emerged 
in the study’s second year. First, many students who had access to programs in the 
second year (53 percent) did not attend. Second, among those who did attend, 
average attendance was low (30 days) and similar to attendance during the first year 
(33 days). 

• Moderate Elementary School Attendance. The first report noted that elementary 
school students attended programs an average of 58 days in the school year. With 
five additional sites and a larger student sample, average attendance was 63 days. 

• Stable Program Leadership, But High Staff Turnover Between the First and 
Second Years. Eighty-two percent of project directors who worked in programs 
during the first year still worked for the programs in the second year. However, two- 
thirds of the line staff and one-third of center coordinators who worked in programs 
during the first year of the study were no longer working for the programs in the 
second year. 



Study Methodology 

The national evaluation of the 21st Century Community Learning Centers program includes an 
elementary school study and a middle school study. 

The elementary school study uses random assignment of students to treatment and control 
groups. The study involved 12 school districts and 26 centers, which were included in the 
evaluation because the centers had more students interested in attending than the centers could 
serve, a precondition for random assignment. The findings are based on baseline and follow- 
up data collected from students, parents, teachers, principals, program staff members, and 
school records. The baseline and follow-up data were collected for 589 treatment group 
students and 384 control group students in 7 school districts in the 2000-2001 school year, and 
for 693 treatment group students and 666 control group students in 5 school districts in the 
2001-2002 school year. The total elementary school sample was 2,308 students. 

The middle school study is based on a nationally representative sample of 21st Century 
programs serving middle school participants and a matched comparison group of students who 
are similar to participants. Similar students were identified in host schools or in other schools 
in the participating districts. Student data were collected from 32 school districts and 61 
centers in those districts. The sample includes 1,782 participants who were matched to 2,482 
comparison students. 

The U.S. Department of Education has funded seven cohorts of grantees. The middle school 
study includes grantees from the first three cohorts of grants, and the elementary school study 
includes grantees from the first five. When the study began, all grantees were in their second 
or third year of a 3-year grant. In 2001, the No Child Left Behind (NCLB) law changed the 
program to state administration; this study does not include 21st Century programs from the 
state-administered program. 

The implementation analysis was based on site visits that were conducted to all grantees, with 
visits lasting between two and four days. Each center was visited twice, once during each of 
the two years of the study. 
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Characteristics and Impacts of Elementary School Programs 



The two most common objectives of administrators of elementary school programs were to 
offer students a safe place after school and to help students improve academically. These goals 
mirror those of parents, who said they enrolled their children in the programs to help them do 
better in school (79 percent of parents) or to provide “a safe place for my child after school” 
(63 percent of parents). 



A Typical Elementary School Center 

The center is open five days a week for three hours a day, serving students in 
kindergarten through grade six. About 120 students a day come to the center. The first 75 
minutes is snack time followed by homework. Certified teachers and aides supervise the 
homework sessions. The next two sessions are 40 minutes each and include academic and 
enrichment activities. For the first session, students alternately work on computers to enhance 
their reading or math skills or meet with a certified teacher for lessons that complement what 
students worked on with computers. For the next session, students are grouped with other 
students in their grade and rotate through enrichment activities such as arts and crafts, karate, 
and fitness and dance. A mix of teachers, instructional aides, and outside organizations lead 
the enrichment activities. On Fridays, students have free choice for one 40-minute block and 
use the time to play board games or basketball. 



Generally programs were open for three hours after school four or five days a week. A 
typical day included one hour for homework and a snack, one hour for another academic activity 
such as a computer lab, and one hour for recreational or cultural activities. 

Eighty-five percent of the centers offered homework assistance, mostly by setting aside time 
for students to do their homework. Eighty-five percent also provided academic activities, such 
as teaching or tutoring, in addition to, or instead of, homework help. 

Moreover, programs provided recreational, cultural, and interpersonal activities. Nearly all 
centers — 92 percent — offered recreational opportunities, ranging from unstructured free time to 
organized sports. Programs also offered dance, drama, and music, and workshops on 
developmental topics, such as building leadership skills and resolving conflicts with peers. 
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Figure 1 

Elementary School Student Attendance 
at Centers 



101 to 175 days 



76 to 1 00 days 



51 to 75 days 



<26 days 




Attendance at programs was 
moderate (Figure 1). Students 
attended, on average, 63 days a year, 
or about two days a week. Almost 
one-fourth of the students attended 
more than 100 days a year and one- 
fourth attended fewer than 26 days. 



Percent of Students 



J Supervision After School. 

26 to 50 days 18.9 Students who attended after-school 

j programs were more likely to be 

<26 days wlth adults who wcrc uot their 

' Percent of Students ~ parents, and less likely to be with 

their parents after school. Students 

[ SOURCE: Center Attendance Records. 

older sibling after school. Programs 
did not affect the frequency with which students reported “self-care,” or the number of days 
when they were at home after school without a parent, another adult, or an older sibling. Just 
over one percent of both groups of students said they were in self-care three or more days in a 
typical week (Figure 2). 



SOURCE: Center Attendance Records. 



Figure 2 

Selected Impacts on Elementary School Students 



Reading test score 
English grade 
Math grade 
Often complete homework 



Self-care 



Not feel safe after school 




‘'Difference is significant at the .05 level. 
SOURCE: Authors’ Calculations, see Chapter II. 



□ Control Group □ Program Group 



Academic Achievement. Students attending after-school programs scored no better on 
reading tests than their peers who did not participate; nor did their grades in English, 
mathematics, science, and social studies increase (Figure 2). In addition, there were no 
statistically significant differences between the two groups of students in time spent on 
homework, student effort in class, preparation for class, and absenteeism; and, according to 
teachers, program students were less likely to complete homework often. 
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Safety After School. Programs improved students’ reported feelings of safety after school; 
1.5 percent of participants, compared with 4.5 percent of nonparticipants, reported feeling “not at 
all safe” after school (Figure 2). 

Developmental Outcomes. Programs had few impacts on developmental outcomes. For 
example, treatment group students were no more likely to report getting along with others their 
age, to rate themselves highly on working with others on a team, or to be able to set goals and 
work to achieve them than nonparticipants. Students in programs were more likely to report 
helping other students after school. 

Negative Behaviors. Students in programs were no less likely than students in the control 
group to be suspended, to receive detention, or to be sent to the office for misbehaving. Students 
in programs were as likely as control students to report negative behaviors, such as breaking 
things, arguing with parents, or giving teachers a hard time. 

Parent Outcomes. Parents of students in programs were more likely to attend after-school 
events in schools. There was no effect on parents attending parent-teacher organization meetings 
or school open houses, or volunteering at school. There was some evidence that programs 
increased whether mothers of elementary students worked or looked for work. Mothers of 
students in programs were more likely to be in the labor force (working full time, part time, or 
looking for work) than mothers of control students. 

Subgroup Impacts, The study looked at subgroup impacts for elementary students but 
found few groups with significant impacts. Students from two-parent households had larger 
impacts on some outcomes than students in single-parent households, but after controlling for 
membership in other subgroups, many of these impacts were no longer significant. 



Characteristics and Outcome Differences of Middle School Programs 

During the second year of the study’s data collection, program administrators indicated that 
their major objectives for programs serving middle school students were to help students 
improve academically and to provide a safe place for them after school. About 80 percent of 
centers offered homework sessions and 60 percent offered other types of academic assistance, 
such as additional help in language arts or mathematics. The emphasis on academics increased 
from the first to second year, according to site visitors, principals, center coordinators, and 
project directors. While our site visit data cannot confirm this shift, there clearly was a 
perception that centers were focusing more on academic activities. 



A Typical 21st Century Middle School Center 

About 45 students participate on an average day. After the school day ends at 
1:30 p.m., students gather in the school cafeteria to get a snack followed by 
homework. After homework time ends, students choose from a variety of activities, 
such as free time in the gym, board games, table tennis, computer lab, and arts and 
crafts. A mix of certified teachers and paraprofessionals supervise the homework 
session and other activities. 
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Programs experienced considerable staff turnover during the 2 years of the study. Two- 
thirds of the staff did not return in the second year; almost one-third of the schools where centers 
were located had a new principal, and one-third had a new center coordinator. Only about 20 
percent of programs had a new project director. Staff most commonly cited the demands on time 
that after-school work posed rather than pay as the reason for not returning. 

Program attendance was much lower in the second year, averaging just 8.8 days. This was 
in large part because many students — 59 percent of the program group — transferred to high 

schools or other middle schools that 
had no 21st Century programs. 
Among the 41 percent of the program 
group who had access to the program 
in the second year, 47 percent attended 
at least 1 day; for the year, their 
attendance averaged 30 days. This is 
similar to the average number of days 
attended in the first year (33 days). 
Ten percent of participating students 
attended more than 75 days and 
59 percent of participating students 
attended fewer than 26 days (Figure 
3). Week-to-week attendance patterns 
also were similar to first-year patterns. 

Supervision After School. The program group was less likely to be with siblings than the 
comparison group. There were no differences in self-care, with roughly 19 percent of 
participants and nonparticipants indicating that they were not with an adult or older sibling three 
or more days a week after school. 

Academic Achievement. There were few differences between the program and comparison 
groups on academic outcomes (Figure 4). The program group had higher grades in social 
studies. Other outcomes — including grades in mathematics, science, and English, as well as 
teacher reports of achievement — did not differ. The level of homework completion also did not 
differ. 

Safety After School. There were no differences between the program group and 
comparison group in feelings of safety after school. 

Developmental Outcomes. The program group was more likely than the comparison 
group — 82 percent versus 79 percent — to expect to graduate from college. No differences were 
observed in other developmental areas. 

Negative Behaviors. Findings on one of several drug-use questions indicated that the 
program group had a higher incidence of drug use (use for both groups was low). There were no 
differences on the other measures of drug use. There were mixed findings on other measures of 
behavior. Treatment students were more likely than comparison students to report breaking 
things on purpose and had higher values on an index of negative behaviors, but there were no 



Figure 3 

Middle School Attendance at Centers in 
the Second Year, Participating Students 




SOURCE: Center Attendance Records. 
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differences on other outcomes such as punching someone, stealing, selling drugs, or getting 
arrested. 

Parent Outcomes. No differences were found in parent involvement. 

Subgroup Impacts. The study examined six subgroups: (1) grade level, (2) whether 

students had low or high reading test scores at baseline, (3) whether students had low or high 
discipline problems at baseline, (4) student race and ethnicity, (5) student gender, and 
(6) whether students lived in two-parent or one-parent households. None showed distinct 
patterns of difference, with one exception: students with low grades (at baseline) had more 
positive impacts than did students with high grades. Reasons for the difference were not clear. 



Comparison of Findings of the First and Second Reports 

The comparison below is presented separately for elementary and middle school students 
because the basis for differences in findings differs for the two groups. For elementary school 
students, differences in findings between the first and second reports are due to the addition of 
new sites to the study; for middle school students, differences in findings relate to an additional 
follow-up year. 



Elementary School Students 

Supervision and Location After School. Both reports found that elementary school 
students attending programs were less likely to be supervised by parents and siblings and more 
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likely to be supervised by other adults. They also were more likely to be at school and less likely 
to be at home during after- school hours. 



Academic Achievement. Both reports found that programs generally did not improve 
academic outcomes such as grades or test scores. In the first report, elementary school students 
had higher grades in social studies but not in English, mathematics, or science. In the second 
report, grades were not higher in any of the four subjects. Both reports found no difference in 
reading test scores. Both reports found homework completion was lower; the second report’s 
finding was statistically significant. 

Safety After School. Both reports found that students reported feeling safer after school; 
only the second report’s finding (based on a larger sample size) was statistically significant. 

Social, Emotional, and Developmental Outcomes. Both reports found that students were 
more likely to help other students after school. There were no differences in other outcomes, 
such as the extent to which students reported getting along with others or setting goals and 
working toward them. 

Negative Behaviors. Students were equally likely to be disciplined for bad behavior, be 
suspended, or receive detention. 

Parent Outcomes. Both reports found that parents were more likely to attend after-school 
events, to help their children with homework, and to ask their children about class. 

Subgroup Outcomes. Neither report found noteworthy patterns of subgroup outcomes. In 
the second report, students from two-parent households had larger impacts on some outcomes 
than students from single-parent households, but these differences were no longer significant 
after controlling for students’ membership in other subgroups. This subgroup was not examined 
in the first report. 



Middle School Students 

Supervision and Location After School. The first report found that program students were 
more likely than comparison-group students to be supervised by other adults and less likely to be 
supervised by parents or siblings. Students also were more likely to be at school and less likely 
to be at home during the after-school hours. In the second report, the only significant findings 
were a reduction in being supervised by siblings and an increase in being at school during the 
after-school hours. 

Academic Achievement. Both reports found few differences in academic outcomes. In the 
first report, students had higher grades in math but not in English, science, or social studies. In 
the second report, students had higher grades in social studies but not in English, math, or 
science. Both reports found no differences in homework completion. School absences were 
lower for treatment students relative to comparison students in both reports. 

Safety After School. Both reports found no differences in feelings of safety after school. 
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Social, Emotional, and Developmental Outcomes. Both reports found an increase in 
students who expected to go to college. 

Negative Behaviors. Both reports found mixed evidence on negative behaviors. Some 
estimates indicated that program students were more likely to engage in negative behaviors and 
others showed no difference. 

Parent Outcomes. The first report indicated that parents were more likely to attend open 
houses, parent/teacher organization meetings, and after-school events, and more likely to 
volunteer at school. The second report found no differences in parent involvement. 

Subgroup Outcomes. The first report found some increases in academic outcomes for 
black and Hispanic middle school students. The second report did not find such increases. 




I. Introduction 



The number of after- school programs has grown quickly in recent years, spurred by rising 
employment rates of mothers, pressures on districts and schools to increase academic 
achievement, concerns about risks to children who are unsupervised in after-school hours, and 
the expansion of federal funding for after-school programs. The percentage of public schools 
offering extended-day programs tripled from 1987 to 1999 (National Center for Education 
Statistics 2002) and estimates from the National Household Education Survey indicate that the 
number of children in kindergarten through 2nd grade in after-school programs grew from 1.6 
million in 1995 to 2.5 million in 2001 (Brimhall and Reaney 1999; Kleiner et al. 2004). 

Eederal funding for after-school programs through the federal 21st Century Community 
Eeaming Centers program rose from $40 million in 1998 to $1 billion in 2002. In addition, 
federal funding from other sources, such as Temporary Aid for Needy Eamilies or the Child Care 
and Development Eund, now supports after-school programs. 

Some research studies have reported that after-school programs increase academic 
achievement, enhance safety, and reduce negative behaviors such as drug and alcohol use 
(Brooks et al. 1995; Hamilton and Klein 1998; Tierney et al. 1995; Welsh et al. 2002). However, 
most studies report negative or neutral findings for some outcomes and positive findings for 
others, patterns that have been noted by observers and researchers reviewing the literature 
(Eashola 1998; Hollister 2003; National Research Council and Institute of Medicine 2002; Roth 
et al. 1998). Some researchers and policymakers have argued that the research base supports 
increased investments in after-school programs (Afterschool Alliance 2003; Eight Crime: Invest 
in Kids 2003; Schwarzenegger 2003), while others have argued the opposite (Olsen 2000). 
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In 1999, the U.S. Department of Education contracted with Mathematica Policy Research, 
Inc., and Decision Information Resources, Inc., to evaluate the 21st Century Community 
Learning Centers program. The program was authorized in the Improving America’s Schools 
Act of 1994 and began awarding grants to school districts in 1998, primarily to support after- 
school programs. 

The evaluation’s elementary school study was based on random assignment, in which 
outcomes of students assigned to the program were compared with outcomes of students who 
were not assigned to the program for lack of space. The evaluation’s middle school study was 
based on a comparison design in which outcomes of students who participated in programs were 
compared with outcomes of similar students who did not. 

In the first year of data collection, school year 2000-2001, researchers collected data for 
roughly 1,000 elementary school students in 18 schools in 7 school districts. In the second year 
of data collection, researchers collected data for five additional grantees, which brought the 
sample up to 2,308 students in 26 schools in 12 districts. Adding five grantees to the study 
enhanced its statistical power for detecting program impacts. This report combines the two 
grantee cohorts and reanalyzes the program’s impacts on elementary school students after 1 year. 
The study is collecting a second year of data for elementary school students, and a future report 
will present findings based on these data. 

The middle school study comprised 4,300 middle school students in 61 schools in 32 school 
districts. Unlike the elementary school study, the middle school study did not add new grantees 
or schools in its second year. In its second year, it gathered more data from students, teachers, 
and parents, which allowed the study to examine outcome differences after two school years. 
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A. Features of the Evaluation Design 

The key features of the evaluation’s design are noted below. Additional information about 
the study design ean be found in chapter I of the first report and in the evaluation’s design report 
(Dynarski et al. 2001). 

Elementary Schools. The evaluation identified 21st Century programs that had waiting lists 
or were turning students away for lack of space and implemented experimental designs. In fall 
2000, roughly 1,000 students from 18 schools in seven grantees applied to 21st Century 
programs and were randomly assigned (findings in the first report were based on this sample). In 
fall 2001, an additional 1,300 students from eight schools in five school districts applied to 21st 
Century programs and were randomly assigned. This new report presents results for the full 
sample of 2,300 students (in 26 elementary schools) after one school year. 

Middle Schools. The evaluation used a comparison design for a nationally representative 
sample of grantees that operated 21st Century programs in middle schools. Thirty-four grantees 
were sampled randomly and agreed to participate in the study, and baseline data were collected 
for 32 grantees (for two grantees, delays in starting data collection were too long to include in the 
report). Students who had attended the program at least three days in a one-month window in 
fall 2000 (according to program records) constituted the study’s “treatment” group. Also in fall 
2000, the study administered questionnaires to more than 21,000 students who were and were not 
participating in 21st Century programs. Using data from the questionnaires, propensity score 
methods matched program students to similar students who were not attending. 

Propensity score methods involved selecting as comparison students those students whose 
characteristics most closely resemble the students in the treatment group on a range of 
demographic and academic characteristics (Rosenbaum and Rubin 1983). Propensity score 
models were estimated separately for each grantee; comparison students were matched only to 
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treatment students in the same school district. In about half of the grantees, comparison students 
were drawn from the same schools attended by participants. In the other half, comparison 
students were drawn from similar schools in the district. 

The matching followed three main steps. First, for each district, the study team estimated a 
logistic regression model in which the dependent variable was participation status and the 
independent variables were student demographic characteristics, indicators of student social 
development, measures of academic performance, and measures of student behavior. (See Table 
B.l in the first report for a listing of matching variables). In most districts, the matching was 
based on 38 student characteristics.^ For participants and potential comparison group students, 
propensity scores were generated based on the estimated models. Second, for each participant, 
we identified the potential comparison group student whose propensity score was numerically 
closest to the participant’s propensity score. To allow for possible attrition in case parent 
consent was not received, we also identified potential comparison group students whose 
propensity scores ranked them as the second- or third-best match. Once we identified matching 
students for each participant, we conducted statistical tests of the equality of the set of 
characteristics for participants and the samples of first-best, second-best, and third-best matches. 
Third, we created an algorithm to generate 2,000 model specifications (created by drawing 
random combinations of characteristics and interactions of characteristics) and carried out steps 
one through four to find the most equivalent comparison groups. 



*In some districts, characteristics had to be dropped from the matching models because they did not vary 
enough. For example, the characteristic for whether students were Hispanic was dropped in some districts that had 
few or no students who were Hispanic. 
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The study used propensity scoring because of its appealing theoretical properties and its 
feasibility. However, in practice, comparison designs generally have lower validity than random 
assignment, a caution that applies to the findings here as well.^ 

Ultimately, about 4,400 students were included in the evaluation’s middle school sample. 
The evaluation collected follow-up data in spring 2001 that formed the basis of the findings 
presented in the first report. It collected another round of follow-up data in spring 2002 that 
forms the basis for the findings presented in this new report. 

Outcomes. The study collected data on a wide array of outcomes that were described in the 
design report and the first report. Outcomes spanned five domains: supervision and location 
after school, academic performance, social and emotional indicators, behavior, and safety. 
Specific outcomes included location and supervision after school, grades, test scores, teachers’ 
perceptions of classroom behavior and effort, school absences, parental involvement, 
victimization, incidents of delinquent behavior, and feelings of safety after school. The wide 
range of outcomes reflects the many objectives embraced by after-school programs. 

Data Sources. The evaluation’s data sources also were described in the first report. They 
include questionnaires completed by students, parents, teachers, principals, and program staff 
members, as well as reading tests, school records, program attendance records, and site visits. 

B. Key Findings From First Report 

The findings from the first year of data collection, which appeared in the February 2003 
report, provide a useful context for results presented in this report. For elementary schools, key 
results were: 



^Rosenbaum and Rubin (1983) show that the propensity scoring technique can be equivalent to random 
assignment under specific conditions. Whether the conditions are met, however, can be verified only in rare 
circumstances. 
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• Students attended programs about 2 days per week, an average of 58 days. 

• After-school program students (who had applied and been assigned to programs) 
were more likely than control-group students to be with an adult who was not their 
parent after school and less likely to be with a parent or sibling. They also were more 
likely to be at school or another place for activities and less likely to be at home after 
school. The incidence of self-care was the same for both groups. 

• Program-group students were as likely as control-group students to report feeling 
safer after school. 

• Program-group students did not improve their reading test scores or their grades in 
math, English, or science relative to the control group. They did improve their grades 
in social studies. 

• Program-group students were as likely as control-group students to complete their 
homework. 

• No relationships were found between impacts and program and student 
characteristics. 



For middle school students, key first- year findings were: 

• Students attended programs about 1 day per week, an average of 33 days. 

• Students attending programs were more likely than control-group students to be with 
an adult who was not their parent after school and less likely to be with a parent or 
sibling. They also were more likely to be at school or another place for activities 
after school and less likely to be in their own homes or the home of someone else. 

• Students attending programs were no more likely to feel safe after school. 

• Students attending programs were not more likely to complete their homework and 
did not improve their grades in English, science, or history, relative to comparison 
students. They did improve their grades in math. 

• Parents of students attending programs were more likely to attend school open 
houses, after-school events, and parent-teacher organization meetings. 

• Students attending programs had higher levels of some negative behaviors, and were 
more likely to be victimized, such as having things taken from them. 



Other recent studies of after-school programs have yielded similar results. For example, 
infrequent attendance has been found for such programs (Grossman et al. 2002; Walker and 
Arbreton 2004), as well as inadequate help with homework (Reisner et al. 2001; Walker and 
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Arbreton 2004), a lack of improvement in grades and test scores (Welsh et al. 2002; Walker and 
Arbreton 2004), and the possibility of more negative behaviors (Sherman et al. 1998). 

However, the findings fall well short of consensus. For example, a recent review of research 
on “out-of-school time” programs (which included summer-school, after-school, and Saturday 
programs) reported that they increased reading and math achievement (Lauer et al. 2003).^ 

C. Report Organization 

The report describes the implementation and impacts of elementary school programs after 
one year of the study, followed by the implementation and outcome differences of middle school 
programs after two years of the study. Because the first report described program 
implementation in detail, this new report focuses on describing key features of the programs’ 
implementation and on identifying areas in which evidence from our implementation study may 
help inform the study’s impact findings. (A future report will assess implementation and impacts 
for elementary school programs after two years of operation.) The appendixes present detailed 
information about response rates and data quality, methods for estimating impacts, and additional 
findings not presented in the main text. 

Two types of additional analyses are presented in the appendix. First, because some middle 
school students graduated or transferred to other schools and did not have access to a 21st 
Century center in the study’s second year, we estimated outcome differences for students who 
had access to centers in the second year. Second, we examined the relationship between center 
attendance and outcomes. Both of these analyses address questions of interest, however, neither 
of the analyses rely on the original matched treatment and comparison groups, therefore, they 

^The review includes findings for intensive programs (such as programs whose purpose was to tutor students in 
reading or math) that differ from those more commonly delivered during out-of-school hours by schools or 
community organizations. 
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provide estimates of lower validity than the estimates presented in the body of the report. For 
this reason, they are presented in the appendix. 
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II. Implementation and Impacts at Elementary School Centers 



The addition of five grantees and more than 1,300 students to the study’s sample allows us 
to reexamine the 21st Century program’s impacts on elementary school students after one school 
year with enhanced statistical power. This chapter provides an overview of the features of 
elementary school centers in the study, focusing on features that may be linked to impacts. It 
then examines student attendance at centers, impacts for the full student sample, and impacts for 
different types of students. 

The estimates show that students attended centers about two days a week on average and 
that the students were more likely to be at school and with adults who were not their parents 
during after-school hours. Students in the control group were more likely to be at home and with 
a parent after school. Centers did not improve student academic achievement as measured by 
homework completion, grades, and reading test scores. These findings generally are consistent 
with findings presented in the first report, which were based on data for seven sites (Dynarski et 
al. 2003). One impact that differs from the first year is that students who attended centers 
reported feeling safer after school."^ 

A. Features of Elementary School 21st Century Centers 

Three features of the 26 elementary school centers in the study are especially useful for 
understanding implementation and impacts: (1) goals and structures of centers, (2) activities and 
service offerings, and (3) characteristics of center staff members.^ Understanding program goals 



■^The first report had similar impact estimates, but the impacts were statistically insignificant. 

^Throughout the report, a “center” refers to after-school services provided in one school, and a “site” refers to 
the group of centers in a school district. A “grantee” refers to a school district that received a 21st Century grant to 
operate centers. A grantee differs from a site because not all centers operated by some grantees were in the study. 
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can inform the impact analysis by highlighting the outcomes that may be affected by centers. If 
an important objective of centers was to improve student academic achievement, assessing 
whether centers improved academic outcomes is appropriate. Similarly, the activities and 
services offered by centers and the types of staff members who work with students are important 
for understanding how centers could affect students. 



1. Center Goals and Structure 

Providing students with a safe place after school and helping them improve academically 
were the two most frequently cited objectives for centers, based on responses to questionnaires 
completed by center administrators. These objectives mirrored the reasons parents most 
frequently gave for having their 
child attend a 21st Century 
center — that the center would “help 
my child do better in school” 

(79 percent of parents) and “it is a 
safe place for my child after 
school” (63 percent of parents). 

Improving relationships between 
schools and parents also was cited 
as a major objective by center 
administrators. 



Percentage of Project Directors Indicating Item 
as One of Three Most Important Objectives 



Provide a Safe, Supervised After-School Environment 66% 

Provide Tutoring/Other Activities to Enhance Students’ 

Ability to Meet Specific Academic Goals 50% 

Provide Academic Enrichment 33% 

Enrich Relationship Between Parents and Schools 25% 

Create a Positive Relationship Between Students and Their 

Schools 16% 

Provide Cultural Opportunities not Available at Home or in 

the Community 16% 

Improve Homework Completion 8% 

Enhance Social Development 8% 

Source: Project Director Survey. Sample size is 12. 



Note: Percentages do not sum to 100 because project directors could 

indicate up to three “most important” objectives. 



(continued) 

For example, some grantees operated centers in both elementary and middle schools, and the study may have 
included only the elementary school centers or only the middle school centers. 
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Centers typically were open during after-school hours for four or five school days a week 



(half were not open on Fridays) and for two to three hours a day. Centers often divided the after- 
school time into roughly hour-long sessions. The first session typically was used for students to 
eat a snack and do their homework. The second session might be for another academic activity, 
such as students’ working on computers or with teachers on their basic skills. The third session 
often would be for development or recreational activities, such as arts and crafts, interpersonal 
skill building, or sports. In three-quarters of the centers, students were required to attend 
academic activities, but typically could choose their activity for the last session of the day or on 
Friday (for centers that were open on Fridays).^ 



2. Activities and Services 

Eighty-five percent of centers offered homework assistance. About half of the centers used 
certified teachers for homework sessions and half used paraprofessionals (one site had certified 
teachers circulate among homework sessions 
monitored by paraprofessionals), with students 
working individually or in small groups. 

Homework help sessions generally were 
unstructured, with students not required to work 
on or complete their homework. Homework 
help was more structured in one-quarter of the 
centers. For example, one center required 
students to complete their homework before they could participate in other activities, and three 



Examples of Academic Activities in 
21st Century Centers 

• Hands-on lessons, such as making exact 
change, solving pre-algebra problems 

• Educational technology packages to 
reinforce basic skills or supplement 
classroom instruction 

• Practice drills in addition, subtraction, 
multiplication, phonics 

• Preparation for standardized tests, such 
as taking and reviewing practice tests, 
completing worksheets related to 
standardized tests 



^At one site, students attended only when they were accompanied by a parent or grandparent, and could choose 
all their activities while the adult participated in technology-oriented instruction. Because the structure of this 
grantee’s program differed from others in the study, we also estimated impacts excluding the grantee and found that 
the main findings were not affected. 
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sites tracked students’ homework assignments. Another center that served children from several 
elementary schools used a homework log that students completed and their classroom teachers 
signed. If students said they did not have homework and their logs were unsigned (which 
occurred frequently), the center faxed their names to their schools to confirm they had no 
homework. Another site also asked teachers to initial a homework log. If students came to the 
center without an initialed homework log, staff members checked the classrooms to see if 
homework assignments had been posted. However, the procedures depended on cooperation 
between after-school center staff members and regular teachers. 

Eighty-five percent of centers also had academic activities, such as teaching or tutoring, in 
addition to, or instead of, homework help. Some centers combined academic activities with 
homework sessions, while other centers set up distinct sessions. Most centers designed their 
academic activities, but some used commercially available packages. Certified teachers usually 
led the activities, sometimes with the help of an aide. Most centers provided activities designed 
to help students improve their reading and math skills. For example, in one academic activity 
observed by a site visitor, the teacher asked 2nd grade students to identify compound words in a 
storybook. In another center, 5th graders used manipulatives to solve pre-algebra problems. In a 
third center, 3rd grade students separated into three groups; while one group worked on reading 
with the aid of a tutor, another group worked independently on math worksheets, and the third 
group worked with a teacher to identify geometric shapes. A few centers helped students prepare 
for standardized tests by giving them practice tests or by working on skills covered by the tests. 

Almost all centers (92 percent) offered recreational activities, which sometimes were 
unstructured — for example, free time, board games, or access to the computer lab. Most centers 
gave students the opportunity to use computers to improve their academic skills or access the 
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Internet to work on school projects. Some recreational activities were more structured, such as 
karate, basketball, and other organized sports that had coaches or instructors. 

Most centers (69 percent) also offered activities to develop interpersonal skills. Activities 
included workshops or discussions on building leadership skills, resolving conflicts, or resisting 
drugs and alcohol. Paraprofessionals or community members typically led these activities. 
Cultural activities, such as arts and music, also were common. 

3. Characteristics of Center Staff 

A mix of certified teachers, paraprofessionals, and community members staffed elementary 
school centers. Centers had an average of 16 paid staff members on their rosters. Center 
coordinators worked an average of four-and-a-half days a week for four hours a day and earned 
just over $19 per hour. Other staff members worked an average of about four days per week for 
three hours a day and earned $15 per hour. The average student-staff ratio across the centers was 
about 7 to 1, ranging from about 4 to 1 to as high as 13 to 1. For most of the staff, the after- 
school job was a second job (71 percent of coordinators and 78 percent of other staff members 
reported that they had another job) and teaching was most often cited as the first job. 

B. Attendance at Centers 

Table II. 1 indicates that students attended 63 days a year, or about 2 days a week (centers 
were open for 30 weeks on average). About one-fourth of students attended centers fewer than 
25 days, half of students attended 26 to 100 days, and one-fourth of students attended more than 
100 days. Almost 60 percent of participants attended less than half the days that centers were 
open. 

’These estimates are based on the total number of students enrolled in a center and the total number of paid 
staff working with students; more precise estimates are difficult because of variations in the number of students and 
staff at a center on a given day. 
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Table II. 1 



21st Century Elementary School Center Attendance, Year 1 



Average Days Attended in School Year “ 


62.5 


Number of Days Attended 


Percent of Students 


1 to 25 Days 


26.4 


26 to 50 Days 


18.9 


51 to 75 Days 


14.8 


76 to 100 Days 


16.4 


101 to 125 Days 


23.5 


Attendance Rate 


Percent of Participants 


10 Percent or less 


19.3 


11 to 25 Percent 


14.5 


26 to 50 Percent 


23.1 


51 to 70 Percent 


19.5 


71 to 85 Percent 


15.4 


86 to 100 Percent 


8.3 



Source: Center Attendance Records. Sample size is 980 students. 

Note: Students who did not attend centers at least one day (19.5 percent of the treatment group) 

are excluded from the table. 

“ Average number of days is calculated for center participants who attended the center at least one 
day after being randomly assigned to the center. Students who did not participate are not included 
in these calculations. 

The attendance rate is the number of days participants attended as a proportion of the number of 
days centers were open, according to grantee annual performance reports. 



Figure II. 1 plots average days attended each week during the school year. The pattern of 
attendance is relatively flat, with sharp dips around major holidays. 
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Figure II. 1 



Days Attended Per Week, Year 1 




Source: Center Attendance Records. 

Note: Students who transferred during the school year are not included in the figure. 



Additional analysis found large differences in average student attendance across grantees. 
For example, one grantee had average student attendance of 44 days a year, whereas another had 

average student attendance of 78 days a year. Variations in average attendance across grantees 

0 

explained much of the variation in student attendance. 

Few student characteristics were related to attendance at centers. We investigated 
15 characteristics, only 3 of which were statistically significant. Students in younger grades 
(grades K through 2), students who were not on public assistance, and students who had not 
moved frequently in the past attended more often. 

C. Impacts of Centers 

Before turning to the impact estimates, it is useful to describe the treatment and control 
groups that are the basis for the estimates. Table II.2 shows that the treatment and control groups 
were similar on a range of characteristics, such as gender, grade level, mother’s age, absences, 

^Models of attendance explained 19 percent of its variation, with 17 percent of the variation in attendance 
related to grantees, and 2 percent related to student characteristics. 
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suspensions, and reading test scores, which is typical when random assignment is used. One 
characteristic differed at the 95 percent confidence level; students in the treatment group were 
more likely to report doing their homework. Considering the large number of variables reported 
in the table, some differences naturally would arise by chance, but the differences do not suggest 
that the treatment group was more or less advantaged than the control group. This same caution 
applies to the impact estimates reported in this chapter; because a large number of statistical tests 
were conducted, some findings may be significant by chance. 

Follow-up response rates were high for student surveys (around 90 percent) but lower for 
parent surveys (75 percent), teacher surveys (79 percent), and student tests (82 percent). The 
study used nonresponse weights to adjust for possible differences in the characteristics of follow- 
up respondents in the treatment and control groups. (See Appendix A for a detailed description 
of the weights.) 

The study also looked at attendance lists to detect whether control-group students attended 
centers. In principle, none of the control-group students would have attended centers. However, 
about nine percent of control-group students were found in the attendance records and, overall, 
the control group averaged 4 days of center attendance (compared to 63 days for the treatment 
group). During the enrollment period, the study detected some crossing over and corrected it. 
However, some crossing over was detected only after time had passed, and the study did not try 
to prevent students who had begun attending from continuing to attend centers. 

Impacts were estimated using regression models to adjust for baseline differences and to 
improve the precision of the estimates. The study also used impact-estimation procedures to 
adjust for crossing over and for treatment group students not attending centers after going 
through random assignment. Appendix B provides details on the procedures used to estimate 
impacts, including the methods used to adjust for crossover by control students and 
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